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ABSTRACT 

This test-retest reliability study was conducted to test the' 
reliability of an instrument designed to measure the relative importance of eight 
implementation conditions. Data for the Ensminger-Surry Implementation Profile 
Instrument (ESIP) were gathered starting January 2003. Participants were from a 
variety of occupations. Ages ranged from the early 20s through the middle 50s. The 
main goal of the ESIP is to determine the relative importance of eight conditions that 
have been shown to facilitate the implementation of an innovation. The testing took 
place in a test-retest format with approximately a 2-week interval between test 
sessions. Participants were graduate students in education in the classes of three 
professors. The study resulted in statistically significant correlations on each of 
the eight dimensions for both forms. Findings show high test-retest reliability for 
one form and fairly strong test-retest reliability for the other form investigated. 
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Reliability Test Results for an Implementation Profile Instrument 

Ovemew 

This paper reports the results of a test - retest reliability study. The study was conducted 
to test the reliability of an instrument designed to measure the relative importance of 
eight implementation conditions. Data for the Ensminger-Surrj' Implementation Profile 
Instrument (ESIPI) were gathered starting in January, 2003. Data collection lasted 
through September, 2003. Participants were from a variety of occupations, ranging from 
elementary education to business management to government. Ages of the participants 
varied from the early 20s through the middle 50s. 

Background 

The main goal of the Ensminger-Surry Implementation Profile Instrument is to determine 
the relative importance of eight conditions that have been shown to facilitate the 
implementation of an innovation. The conditions were developed by Donald P. Ely (Ely 
1990, 1999; Suny &. Ely, 2001). These conditions are: 

1. ) Dissatisfaction with the status quo, 

2. ) Knowledge and Skills, 

3. ) Adequate Resources, 

4. ) Availability of Time, 

5. ) Rewards and Incentives, 

6. ) Participation and Involvement, 

7. ) Commitment and Support, and 

8. ) Leadership. 
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The purpose of the ESIPI is to determine which conditions are more or less important in 
an individual’s decision to implement an innovation. There are two versions of ESIPI: 
one determines the relative importance of the eight conditions related to a process 
innovation and one determines the relative importance of the eight conditions related to a 
technology innovation. By determining the relative importance of each condition, an 
individual’s “implementation profile” can be determined. Also, by determining the 
profile for everyone within an organization, it will be possible to determine an 
“organizational implementation profile.” 

Instrument 

The ESIPI is a 56-item instrument. Each item presents a statement representing two 
implementation conditions. For example, the instrument may pair a statement related to 
the condition of “Time” with one related to the condition of “Leadership.” Participants 
must then choose which condition is more important to their decision to implement an 
innovation. Seven outside experts familiar with innovation and change validated the 
statements presented on the instrument. Each condition is paired with each of the other 
seven conditions twice. This results in the 56 items on the instrument. A person can score 
from 0 to 14 on each of the eight conditions, depending upon the number of times they 
selected that condition. A person’s “implementation profile” is, therefore, represented by 
a score from 0 to 14 on each condition. 
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Participants 

Thirty-nine participants completed both the test and retest for the technology form. 
Thirty-eight participants completed both the test and retest for process form. None of 
the participants completed both forms. The nature of this study did not allow for strict 
control of the test environment. The reliability testing occurred in a test-retest format with 
approximately a 2 week interval between test adminstration sessions. It took 
approximately 20 minutes to complete the form each time. The participants were a 
mixed sample taken from graduate level classes in Education. All of the participants had 
at least an undergraduate education. There was a wide variety in ethnicity, age, 
occupation, gender, educational level, and familiarity with technology. 

Environment 

Three professors agreed to allow the test to be given on a voluntary basis to students in 
each of their classes. The testing took place before the start of designated class times. 
Each testing period took place in the classroom without distraction. Special arrangements 
were made for those participants with disabilities who wished to participate. Participants 
were informed of the nature of the study as well as any potential risks or benefits. Written 
consent was provided by each participant prior to the first test. 

Results 

Pearson’s r was used to determine the relationship between the profile scores on the first 
test and profile scores on the retest. Prior to analyzing our data, we decided that a .70 
correlation coefficient would be the standard for strong reliability. 




4 



5 



Thirty-nine pairs of participant responses were included in the statistical analysis of 
technology form. The correlations for all eight conditions were statistically significant 
(see Table 1). The categories of Skill and Knowledge, Availability of Time, Participation 
and Involvement, Commitment and Support, and Leadership all showed a strong test- 
retest correlation above the .70 standard. Three categories (Dissatisfaction with Status 
Quo, Adequate Resources, and Rewards and Incentives) resulted in statistically 
significant correlations, albeit slightly below the .70 standard. The mean correlation for 
technology form was .747. 

Table 1: Test/Retest Coefficients and P-Values technology form (N=39) 



Conditions 


Correlation 
Pretest/ Posttest 


P- 

Value 


Dissatisfaction with the Status Quo 


.586 


.05 


Knowledge and Skills 


.761 


.01 


Availability of Resources 


.687 


.01 


Availability of Time 


.773 


.01 


Rewards and Incentives 


.646 


.01 


Participation 


.864 


.01 


Commitment 


.782 n 


.01 


Leadership 


.778 


.01 


MEAN 


.747 


- 



Thirty-eight usable pairs of participant responses were included in the statistical analysis 
of the process form. The correlations for all eight conditions were statistically significant 
(see Table 2). The categories of Rewards and Incentives, Availability of Time, 
Participation and Leadership all showed a strong test-retest correlation at or above the .70 
standard. The remaining four categories (Dissatisfaction with Status Quo, Adequate 
Resources, Skills and Knowledge, and Commitment) resulted in statistically significant 
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correlations, albeit below the .70 standard. The mean correlation for process form was 
.612. 



Table 2: Test/ Retest Coefficients and P-Values process form 0^=38) 



Conditions 


Correlation 
Pretest/ Posttest 


P- 

Value 


Dissatisfaction with the Status Quo 


.349 


.032 


Knowledge and Skills 


.618 


.000 


Availability of Resources 


.478 


.002 


Availability of Time 


.764 


.000 


Rewards and Incentives 


.804 


.000 


Participation 


.695 


.000 


Commitment 


.439 


.006 


Leadership 


.746 


.000 


MEAN 


.612 


- 



In summary, the test-retest reliability study of the Ensminger-Suriy Implementation 
Profile Instrument resulted in statistically significant correlations on each of the eight 
conditions for both forms. The mean correlation coefficient for technology form was 
.747. We believe this shows there is extremely high test-retest reliability for this form of 
the instrument. The mean correlation coefficient for Form B was .612. We believe this 
shows there is fairly strong test-retest reliability for this form of the instrument. More 
research, including individual item analysis, is needed to determine the cause of the 
different levels of reliability of the 2 forms of the instrument. 
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